How would you benchmark whether a proposed quantization configuration is worth the accuracy trade-off for your workload?

A quantization benchmark has three outputs: how much memory it saves, how much latency it changes, and how much recall it costs. The memory and latency parts are easy - compare the collection's reported size and measure p50/p99 latency at a fixed query load. The recall part is the one teams get wrong, because it requires ground truth. The correct method is to run an exact search (brute force, no quantization, no HNSW approximation) over the same queries on the same corpus and treat those results as ground truth. Then run the same queries against the quantized collection and compute recall@k as the fraction of ground-truth top-k that appear in the quantized collection's top-k. That number, together with the latency and memory deltas, is the trade-off. Recall must be measured with the exact configuration you plan to ship, including oversampling and rescoring, because those settings materially change the result.

The most important methodological detail is the query set. Using random vectors or a small synthetic query set will give misleading results because quantization error is distribution-dependent - it depends on how the data is clustered and how close the near neighbors are. The benchmark must use queries drawn from the actual production distribution, ideally a few thousand of them, so that the recall estimate has acceptable variance. If you cannot get production queries, use a held-out set from the same source as your corpus, not random noise. The second detail is to control for confounding variables. If you change quantization and also change ef, you cannot attribute the recall difference to quantization. Hold ef, m, and the candidate set sizes fixed across configurations, and vary only the quantization settings. The third detail is to measure at the operating point you will use - if you plan to raise oversampling to 5 to recover recall, benchmark at oversampling 5, not at the default.

The trade-off being evaluated is always the same: memory and latency against recall. There is no universal answer because the acceptable recall loss depends on the application - a recommendation system may tolerate a 2-point recall drop, while a compliance search may not. The benchmark's job is to produce the curve so the product owner can choose a point on it. The common mistake is measuring recall against another approximate configuration rather than exact ground truth, which makes both numbers wrong and can hide a large regression. The second mistake is benchmarking on a static snapshot and shipping, then discovering that recall degrades as the corpus drifts - quantization error is data-dependent, so re-benchmark periodically. The third mistake is ignoring p99: a configuration that looks fine at p50 can have a much worse tail because rescoring touches disk. Version note: the available quantization schemes, the way oversampling and rescoring are configured, and the collection's reported memory usage have all changed across Qdrant releases - benchmark on the version you will deploy, not on a different one.

Version-dependent: the QuantizationSearchParams fields (rescore, oversampling) and the available quantization schemes differ across releases, and the collection's reported memory usage is not directly comparable between versions because the internal layout changed. If you are comparing configurations, do it on a single version and re-benchmark after upgrades rather than comparing numbers across versions.